I looked into this more. Here is the information I found:

It is possible to get a pointer/handle from a awt component on OSX. MozSwing is actually using this as it starts with a awt.Canvas component, gets the pointer, and passes that to xulrunner. I haven't looked at that code closely to figure out what it is doing. Michal can probably summarize how it works.

I believe it is using the getViewPtr() method that is available on the peers of awt components on OSX. I assume this viewPtr corresponds to the NSView of the awt component: http://developer.apple.com/documentation/Cocoa/Reference/ApplicationKit/Classes/NSView_Class/Reference/NSView.html

Cocoa supports clipping. Here is some documentation on that: http://developer.apple.com/documentation/Cocoa/Conceptual/CocoaDrawingGuide/GraphicsContexts/chapter_3_section_3.html#//apple_ref/doc/uid/TP40003290-CH203-BCIIADBC

I'm skeptical that simply setting a clip region on the awt.graphics object will have any affect on how cocoa components are drawn. I looked a bit at the WindowUtils class, and found on OSX it is getting the peer of the window and using reflection to call a setAlpha method. So basically it is making the native window be transparent, that way if only swing components are used inside of the window then the clipping applied by swing will be respected. This type of a approach would probably work for clipping a simple canvas implementation that uses awt.graphics to do all of its drawing. The background color of the canvas can set to be transparent and the graphics passed to the paint method of the canvas can be clipped.
However we want to support native components and on OSX that probably means NSViews. These components draw in the drawRect method by finding their current graphics context like this:
NSGraphicsContext currentContext
if that is called inside of the drawRect the returned currentContext is clipped and the coordinates are transformed.

So in the case of mozswing we can probably patch xulrunner to apply a clipping before it does the drawing with the NSGraphicsContext. But that won't be a general solution for native components.

In a few places in the Cocoa documentation it mentions superviews can clip subviews, but I haven't seen how to do that. If that is possible then any native component that uses a NSView could be embedded in a superview and the superview could apply the clipping that NativeSwing figures out.

Another option is that it sounds like cocoa automatically applies clipping to sub views when they are behind opaque sibling sub views. So we might be able to add fake sibling sub views that say they are opaque but actually they don't draw anything.

I'm going to keep poking around a little more, and then I'll hand it off to Dima (our mac Java/Cocoa guy), unless Michal wants to try working on it.

Scott

Christopher Deckers wrote:
> Scott, Michal,
>
> As you know, I have more information about the clipping issue on Mac
> OS, but I have not found any working solution. The main issue is that
> I don't have a Mac to test my ideas.
> So here is what I know and a few ideas in case someone could continue
> the investigation.
>
> First, I attach a really simple test case which works on Windows and
> Linux. A Canvas (a heavyweight AWT component) is added to a JFrame,
> and it is clipped to the intersection of 2 rectangles. I use JNA and a
> modified version of its WindowUtils class that allows to clip to the
> intersection of rectangles (this modified version can be found at
> http://djproject.cvs.sourceforge.net/*checkout*/djproject/jna_WindowUtils/src/com/sun/jna/examples/WindowUtils.java
> ).
>
> There are several things to try:
> 1. A similar approach to the one of Linux and Windows.
> 2. Creating a subclass of a native component that tweaks its paint
> method, and which contains the native component.
> 3. Adding an AWT Panel which contains a JPanel subclass that tweaks
> its paint method, and which itself contains our native component
> (generally a Canvas).
> 4. Any other approach?
>
> 1. On Windows and Linux, a native handle is obtained from the AWT
> heavyweight component. This native handle allows to call native APIs
> (using JNA) that sets a region clip. The problem with this approach on
> the Mac is that it seems there is no such concept as a handle in
> Cocoa.
>
> 2. Tweaking the paint method seems to be a correct approach: I was
> told that there is no real lightweight and heavyweight difference. It
> only consists of some tricks at the Graphics level. The initial tests
> that I gave to Michal were a quick attempt, by replicating the kind of
> code found in JNA for its Window trick. This failed, but I don't know
> the reason so I can't tell whether the approach is incorrect or if I
> made a mistake somewhere.
>
> 3. JNA implements clipping for AWT Windows only on Mac OS: the AWT
> window contains a JPanel subclass which tweaks its paint method, and
> this JPanel contains the rest of the interface. It seems that tweaking
> the paint method is enough to have an effect on the outer AWT
> component.
> The code for this Window clipping on Mac OS can be found in this
> WindowUtils class in an inner class named OSXTransparentContent. This
> inner class is a JPanel subclass that modifies the paint(Graphics)
> method to apply some clipping.
>
> 4. Well, maybe having a look at what Cocoa is able to do could help
> give more ideas.
>
> I would suggest trying #3 first, as it is the solution more likely to work.
>
> Please let me know if someone is going to give it a try, and feel free
> to contact me if you want more info.
>
> Cheers,
> -Christopher
>